Gaze-based Attention Recognition for Human-Robot Collaboration

Attention (and distraction) recognition is a key factor in improving human-robot collaboration. We present an assembly scenario where a human operator and a cobot collaborate equally to piece together a gearbox. The setup provides multiple opportunities for the cobot to adapt its behavior depending on the operator's attention, which can improve the collaboration experience and reduce psychological strain. As a first step, we recognize the areas in the workspace that the human operator is paying attention to, and consequently, detect when the operator is distracted. We propose a novel deep-learning approach to develop an attention recognition model. First, we train a convolutional neural network to estimate the gaze direction using a publicly available image dataset. Then, we use transfer learning with a small dataset to map the gaze direction onto pre-defined areas of interest. Models trained using this approach performed very well in leave-one-subject-out evaluation on the small dataset. We performed an additional validation of our models using the video snippets collected from participants working as an operator in the presented assembly scenario. Although the recall for the Distracted class was lower in this case, the models performed well in recognizing the areas the operator paid attention to. To the best of our knowledge, this is the first work that validated an attention recognition model using data from a setting that mimics industrial human-robot collaboration. Our findings highlight the need for validation of attention recognition solutions in such full-fledged, non-guided scenarios.Comment: Accepted to PETRA 202

Towards social embodied cobots: The integration of an industrial cobot with a social virtual agent

The integration of the physical capabilities of an industrial collaborative robot with a social virtual character may represent a viable solution to enhance the workers' perception of the system as an embodied social entity and increase social engagement and well-being at the workplace. An online study was setup using prerecorded video interactions in order to pilot potential advantages of different embodied configurations of the cobot-avatar system in terms of perceptions of Social Presence, cobot-avatar Unity and Social Role of the system, and explore the relation of these. In particular, two different configurations were explored and compared: the virtual character was displayed either on a tablet strapped onto the base of the cobot or on a large TV screen positioned at the back of the workcell. The results imply that participants showed no clear preference based on the constructs, and both configurations fulfill these basic criteria. In terms of the relations between the constructs, there were strong correlations between perception of Social Presence, Unity and Social Role (Collegiality). This gives a valuable insight into the role of these constructs in the perception of cobots as embodied social entities, and towards building cobots that support well-being at the workplace

The effects of robotic assistance on upper limb spatial muscle synergies in healthy people during planar upper-limb training

BackgroundRobotic rehabilitation is a commonly adopted technique used to restore motor functionality of neurological patients. However, despite promising results were achieved, the effects of human-robot interaction on human motor control and the recovery mechanisms induced with robot assistance can be further investigated even on healthy subjects before translating to clinical practice. In this study, we adopt a standard paradigm for upper-limb rehabilitation (a planar device with assistive control) with linear and challenging curvilinear trajectories to investigate the effect of the assistance in human-robot interaction in healthy people.MethodsTen healthy subjects were instructed to perform a large set of radial and curvilinear movements in two interaction modes: 1) free movement (subjects hold the robot handle with no assistance) and 2) assisted movement (with a force tunnel assistance paradigm). Kinematics and EMGs from representative upper-limb muscles were recorded to extract phasic muscle synergies. The free and assisted interaction modes were compared assessing the level of assistance, error, and muscle synergy comparison between the two interaction modes.ResultsIt was found that in free movement error magnitude is higher than with assistance, proving that task complexity required assistance also on healthy controls. Moreover, curvilinear tasks require more assistance than standard radial paths and error is higher. Interestingly, while assistance improved task performance, we found only a slight modification of phasic synergies when comparing assisted and free movement.ConclusionsWe found that on healthy people, the effect of assistance was significant on task performance, but limited on muscle synergies. The findings of this study can find applications for assessing human-robot interaction and to design training to maximize motor recovery

Biomechanical Assessments of the Upper Limb for Determining Fatigue, Strain and Effort from the Laboratory to the Industrial Working Place: A Systematic Review

Recent human-centered developments in the industrial field (Industry 5.0) lead companies and stakeholders to ensure the wellbeing of their workers with assessments of upper limb performance in the workplace, with the aim of reducing work-related diseases and improving awareness of the physical status of workers, by assessing motor performance, fatigue, strain and effort. Such approaches are usually developed in laboratories and only at times they are translated to on-field applications; few studies summarized common practices for the assessments. Therefore, our aim is to review the current state-of-the-art approaches used for the assessment of fatigue, strain and effort in working scenarios and to analyze in detail the differences between studies that take place in the laboratory and in the workplace, in order to give insights on future trends and directions. A systematic review of the studies aimed at evaluating the motor performance, fatigue, strain and effort of the upper limb targeting working scenarios is presented. A total of 1375 articles were found in scientific databases and 288 were analyzed. About half of the scientific articles are focused on laboratory pilot studies investigating effort and fatigue in laboratories, while the other half are set in working places. Our results showed that assessing upper limb biomechanics is quite common in the field, but it is mostly performed with instrumental assessments in laboratory studies, while questionnaires and scales are preferred in working places. Future directions may be oriented towards multi-domain approaches able to exploit the potential of combined analyses, exploitation of instrumental approaches in workplace, targeting a wider range of people and implementing more structured trials to translate pilot studies to real practice

Kinect V2 Performance Assessment in Daily-Life Gestures: Cohort Study on Healthy Subjects for a Reference Database for Automated Instrumental Evaluations on Neurological Patients

Background. The increase of sanitary costs related to poststroke rehabilitation requires new sustainable and cost-effective strategies for promoting autonomous and dehospitalized motor training. In the Riprendo@Home and Future Home for Future Communities research projects, the promising approach of introducing low-cost technologies that promote home rehabilitation is exploited. In order to provide reliable evaluation of patients, a reference database of healthy people’s performances is required and should consider variability related to healthy people performances. Methods. 78 healthy subjects performed several repetitions of daily-life gestures, the reaching movement (RM) and hand-to-mouth (HtMM) movement with both the dominant and nondominant upper limbs. Movements were recorded with a Kinect V2. A synthetic biomechanical protocol based on kinematical, dynamical, and motor control parameters was used to assess motor performance of the healthy people. The investigation was conducted by clustering participants depending on their limb dominancy (right/left), gender (male/female), and age (young/middle/senior) as sources of variability. Results. Results showed that limb dominancy has minor relevance in affecting RM and HtMM; gender has relevance in affecting the HtMM; age has major effect in affecting RM and HtMM. Conclusions. An investigation of healthy subjects’ upper limb performances during daily-life gestures was performed with the Kinect V2 sensor. Findings will be the basis for a database of normative data for neurological patients’ motor evaluation

Robotic Assistance for Upper Limbs May Induce Slight Changes in Motor Modules Compared With Free Movements in Stroke Survivors: A Cluster-Based Muscle Synergy Analysis

Background: The efficacy of robot-assisted rehabilitation as a technique for achieving motor recovery is still being debated. The effects of robotic assistance are generally measured using standard clinical assessments. Few studies have investigated the value of human-centered instrumental analysis, taking the modular organization of the human neuromotor system into account in assessing how stroke survivors interact with robotic set-ups. In this paper, muscle synergy analysis was coupled with clustering procedures to elucidate the effect of human-robot interaction on the spatial and temporal features, and directional tuning of motor modules during robot-assisted movements.Methods: Twenty-two stroke survivors completed a session comprising a series of hand-to-mouth movements with and without robotic assistance. Patients were assessed instrumentally, recording kinematic, and electromyographic data to extract spatial muscle synergies and their temporal components. Patients' spatial synergies were grouped by means of a cluster analysis, matched pairwise across conditions (free and robot-assisted movement), and compared in terms of their spatial and temporal features, and directional tuning, to examine how robotic assistance altered their motor modules.Results: Motor synergies were successfully extracted for all 22 patients in both conditions. Seven clusters (spatial synergies) could describe the original datasets, in both free and robot-assisted movements. Interacting with the robot slightly altered the spatial synergies' features (to a variable extent), as well as their temporal components and directional tuning.Conclusions: Slight differences were identified in the characteristics of spatial synergies, temporal components and directional tuning of the motor modules of stroke survivors engaging in free and robot-assisted movements. Such effects are worth investigating in the framework of a modular description of the neuromusculoskeletal system to shed more light on human-robot interaction, and the effects of robotic assistance and rehabilitation

Proprioceptivity and Upper-Extremity Dynamics in Robot-Assisted Reaching Movement

Abstract-Reaching-against-gravity movements feature some remarkable aspects of human motion, like a wide exploration of the upper extremity workspace and high dynamics. In clinical rehabilitation protocols the recovery of the reaching movement capability is considered as a "paradigm" because of its fundamental role as a precursor for the use of the hand in activities of daily living. Reaching-based protocol may take advantage of robot usage, which has become a standard procedure in rehabilitation of neurological patients although the efficacy of the robot-assisted treatment is still matter of discussion. Even fewer studies in literature investigate proprioception, upper-extremity dynamics and their mutual relationship. Robot-assistance introduces alterations in the dynamics of movements, e.g. limited maximum velocities and accelerations, partial upper-extremity weight support, interaction forces between the robot and a subject. As a consequence, the subjects' proprioception may be altered too. The purpose of this preliminary work is to investigate the relationship between upper-extremity dynamics and proprioception by comparing the estimation of shoulder torques and EMG activation pattern with the evaluation given by the subjects on the quality of the perceived movements during different reaching trials with and without robot assistance. Results show that slow free (nonassisted) reaching movements are felt as uncomfortable and figure large shoulder torques and EMG cocontraction levels. Comfortable movements are those displaying shoulder torques and cocontraction levels comparable to those in natural free reaching, suggesting the strong correlation of torques patterns and co-contractions in motion comfort

On a Two-DoF Parallel and Orthogonal Variable-Stiffness Actuator: An Innovative Kinematic Architecture

Variable-Stiffness Actuators are continuously increasing in importance due to their characteristics that can be beneficial in various applications. It is undisputed that several one-degree-of-freedom (DoF) solutions have been developed thus far. The aim of this work is to introduce an original two-DoF planar variable-stiffness mechanism, characterized by an orthogonal arrangement of the actuation units to favor the isotropy. This device combines the concepts forming the basis of a one-DoF agonist-antagonist variable-stiffness mechanism and the rigid planar parallel and orthogonal kinematic one. In this paper, the kinematics and the operation principles are set out in detail, together with the analysis of the mechanism stiffness

Optimal Biomechanical Performance in Upper-Limb Gestures Depends on Velocity and Carried Load

In the last few years, there has been increased interest in the preservation of physical and mental health of workers that cooperate with robots in industrial contexts, such as in the framework of the European H2020 Mindbot Project. Since biomechanical analysis contributes to the characterization of the subject interacting with a robotic setup and platform, we tested different speed and loading conditions in a simulated environment to determine upper-limb optimal performance. The simulations were performed starting from laboratory data of people executing upper-limb frontal reaching movements, by scaling the motion law and imposing various carried loads at the hand. The simulated velocity ranged from 20% to 200% of the original natural speed, with step increments of 10%, while the hand loads were 0, 0.5, 1, and 2 kg, simulating carried objects. A 3D inverse kinematic and dynamic model was used to compute upper-limb kinematics and dynamics, including shoulder flexion, shoulder abduction, and elbow flexion. An optimal range of velocities was found in which the expended energy was lower. Interestingly, the optimal speed corresponding to lower exerted torque and energy decreased when the load applied increased. Lastly, we introduced a preliminary movement inefficiency index to evaluate the deviation of the power and expended energy for the shoulder flexion degree of freedom when not coinciding with the minimum energy condition. These results can be useful in human–robot collaboration to design minimum-fatigue collaborative tasks, tune setup parameters and robot behavior, and support physical and mental health for workers

A Planar Parallel Device for Neurorehabilitation

The patient population needing physical rehabilitation in the upper extremity is constantly increasing. Robotic devices have the potential to address this problem, however most of the rehabilitation robots are technically advanced and mainly designed for clinical use. This paper presents the development of an affordable device for upper-limb neurorehabilitation designed for home use. The device is based on a 2-DOF five-bar parallel kinematic mechanism. The prototype has been designed so that it can be bound on one side of a table with a clamp. A kinematic optimization was performed on the length of the links of the manipulator in order to provide the optimum kinematic behaviour within the desired workspace. The mechanical structure was developed, and a 3D-printed prototype was assembled. The prototype embeds two single-point load cells to measure the force exchanged with the patient. Rehabilitation-specific control algorithms are described and tested. Finally, an experimental procedure is performed in order to validate the accuracy of the position measurements. The assessment confirms an acceptable level of performance with respect to the requirements of the application under analysis